Skip to content

Lower threshold on backoff for sanity tests #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 23, 2022
Merged

Lower threshold on backoff for sanity tests #1087

merged 1 commit into from
Nov 23, 2022

Conversation

judemars
Copy link
Contributor

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation

/kind failing-test

/kind feature
/kind flake

What this PR does / why we need it:
The scenario of the failing test ("should fail when the volume is already published but is incompatible"):

  1. We create a volume -> succeeds
  2. We publish that volume -> succeeds
  3. We re-publish that volume, expecting an error bc it was already published -> fails (and triggers backoff)
  4. Cleanup function tries to unpublish volume, and this fails the test case with the error ControllerUnpublish not permitted on node "projects/test-project/zones/country-region-zone/instances/test-name" due to backoff condition because of Step 3.

So, the solution we decided in #990 was to lower the duration of backoff for sanity tests so that the backoff triggered in step 3 above will not cause step 4 to fail.

See #990.

Which issue(s) this PR fixes:

Fixes #990

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 21, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @judemars. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 21, 2022
@judemars judemars mentioned this pull request Nov 21, 2022
Closed
saikat-royc
saikat-royc reviewed Nov 22, 2022
View reviewed changes
cmd/gce-pd-csi-driver/main.go Outdated
@@ -57,6 +57,8 @@ var (

maxprocs = flag.Int("maxprocs", 1, "GOMAXPROCS override")

errorBackoffInitialDuration = 200 * time.Millisecond
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make this a flag for errorBackoffMaxDuration and errorBackoffInitialDuration, with default values

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! Done

Copy link
Member

@saikat-royc saikat-royc Nov 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also make the errorBackoffMaxDuration configurable through a separate flag

Copy link
Contributor Author

@judemars judemars Nov 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woops sorry I missed that, added errorBackoffMaxDurationMs flag. Note I did it in ms to match the initial duration flag, but lmk if you think I should change to minutes/seconds.

@saikat-royc
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 22, 2022
saikat-royc
saikat-royc reviewed Nov 22, 2022
View reviewed changes
cmd/gce-pd-csi-driver/main.go Outdated
@@ -42,7 +42,8 @@ var (
httpEndpoint = flag.String("http-endpoint", "", "The TCP network address where the prometheus metrics endpoint will listen (example: `:8080`). The default is empty string, which means metrics endpoint is disabled.")
metricsPath = flag.String("metrics-path", "/metrics", "The HTTP path where prometheus metrics will be exposed. Default is `/metrics`.")

extraVolumeLabelsStr = flag.String("extra-labels", "", "Extra labels to attach to each PD created. It is a comma separated list of key value pairs like '<key1>=<value1>,<key2>=<value2>'. See https://cloud.google.com/compute/docs/labeling-resources for details")
errorBackoffInitialDurationMs = flag.Int("backoff-initial-duration-ms", 200, "The amount of ms for the initial duration of the backoff condition. Default is 200.")
Copy link
Member

@saikat-royc saikat-royc Nov 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... backoff condition for controller publish/unpublish CSI operations

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, thanks

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 22, 2022
@amacaskill
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 22, 2022
@judemars judemars mentioned this pull request Nov 23, 2022
Closed
9 tasks
@mattcary
Copy link
Contributor

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: judemars, mattcary

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 23, 2022
@k8s-ci-robot k8s-ci-robot merged commit 3fdf68f into kubernetes-sigs:master Nov 23, 2022
@judemars judemars mentioned this pull request Dec 1, 2022
Merged
scbrubaker02
scbrubaker02 reviewed Dec 5, 2022
View reviewed changes
cmd/gce-pd-csi-driver/main.go
@@ -122,7 +124,9 @@ func handle() {
if err != nil {
klog.Fatalf("Failed to get cloud provider: %v", err)
}
controllerServer = driver.NewControllerServer(gceDriver, cloudProvider)
initialBackoffDuration := time.Duration(*errorBackoffInitialDurationMs) * time.Millisecond
maxBackoffDuration := time.Duration(*errorBackoffMaxDurationMs) * time.Microsecond
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judemars, looks like this should be time.Millisecond.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Sorry about that! Assigned #1098 to you.

@judemars judemars mentioned this pull request Dec 5, 2022
Merged
@leiyiz leiyiz mentioned this pull request Jan 18, 2023
Merged
k8s-ci-robot added a commit that referenced this pull request Jan 18, 2023
@k8s-ci-robot
Merge pull request #1111 from leiyiz/release-1.8
1751513 
cherry pick #1085 #1087 and #1094 to release-1.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scbrubaker02 scbrubaker02 scbrubaker02 left review comments

@saikat-royc saikat-royc saikat-royc left review comments

@amacaskill amacaskill Awaiting requested review from amacaskill

@mattcary mattcary Awaiting requested review from mattcary

Assignees

@amacaskill amacaskill

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Sanity test do not work with the PDCSI driver attach detach behavior change
6 participants
@judemars @k8s-ci-robot @saikat-royc @amacaskill @mattcary @scbrubaker02